Deep parameter-free attention hashing for image retrieval

Deep hashing method is widely applied in the field of image retrieval because of its advantages of low storage consumption and fast retrieval speed. There is a defect of insufficiency feature extraction when existing deep hashing method uses the convolutional neural network (CNN) to extract images semantic features. Some studies propose to add channel-based or spatial-based attention modules. However, embedding these modules into the network can increase the complexity of model and lead to over fitting in the training process. In this study, a novel deep parameter-free attention hashing (DPFAH) is proposed to solve these problems, that designs a parameter-free attention (PFA) module in ResNet18 network. PFA is a lightweight module that defines an energy function to measure the importance of each neuron and infers 3-D attention weights for feature map in a layer. A fast closed-form solution for this energy function proves that the PFA module does not add any parameters to the network. Otherwise, this paper designs a novel hashing framework that includes the hash codes learning branch and the classification branch to explore more label information. The like-binary codes are constrained by a regulation term to reduce the quantization error in the continuous relaxation. Experiments on CIFAR-10, NUS-WIDE and Imagenet-100 show that DPFAH method achieves better performance.

in Fig. 1, the whole process is mainly divided into four steps. First, the pairs of images are fed into the Convolution layer and the Maximum pooling layer to generate feature map. Second, the feature map is processed by PFA module, which considers both the 1-D channel-wise weights and the 2-D spatial-wise weights and directly generated 3-D weights. Third, this paper performs the operation of element-wise sum on PFA output and feature map and input the result to the backbone network to extract image features. Finally, in order to make efficient use of semantic label information, two branches containing classifier layer and hashing layer is designed. Combining the pair-wise loss and quantization loss generated by the hashing layer and class-wise loss generated classifier to obtain hash codes with discriminative ability.
In short, the contributions are as follows: 1. DPFAH is an end-to-end learning framework which perform simultaneous feature representation and binary codes learning. A lightweight module is introduced to extract rich semantic features and avoid over fitting in the training process.
2. The PFA module is embedded in ResNet18 network to improve the feature representation. It explores an energy mathematical formula to calculate the 3-D weight and derives a closed-form solution that speedup the weight calculation. No parameters are added to the network during the whole process. 3. A novel deep hashing framework is designed by DPFAH, which includes hashing learning and classification. This method can use the label information to eliminate discrepancy and generate more accurate hash codes. Experimental results on three datasets have verified DPFAH.
The remaining content of this paper is as follows. "Related work" is related work. "Deep Parameter-free attention hashing" describes the details of DPFAH. "Experiments" is the results of experiments and analysis. "Conclusions" summarizes the work of this study.

Related work
Deep learning is applied in many fields for its advantages of a solid learning ability and good portability. Network security fields use neural networks to detect malware 22,23 and programs 24 , The field of artificial intelligence can be conducive to the intelligent estimation of traffic time by deep learning methods 25 . This paper focuses on the research of hashing algorithm based on deep learning. Deep hashing is widely applied in image retrieval system due to its own advantages. For example, the function of searching images by image is realized through deep hashing in many shopping software. Therefore, how to obtain hash code with strong accuracy for each image has become a research hotspot. In this section, the existing several unsupervised hashing approaches and supervised hashing approaches are introduced.
Unsupervised hashing. Unsupervised hashing [26][27][28][29][30] only utilizes the unlabeled data points to learn hash function that map high dimensional feature to compact hash codes. The similarity matrix is usually constructed in the process of feature learning. Many scholars have carried a lot of study on the perspective of constructing similarity matrix. Specifically, Sheng et al. proposed 28 the descriptors of data are represented by the output of full-connected layer and used to design the similarity matrix. The network is optimized by calculating the loss between the similarity matrix and pairwise hash codes. By observing the law of features distribution, Yang et al. proposed 29 the cosine distance of pairs data can be evaluated by Gaussian distributions. They set a distance threshold in the steps of constructing the similarity matrix, the data points are defined as similar if the cosine distance of data points smaller than threshold, vice versa. On this basis, Jiang et al. proposed 30 the cosine distance was used directly to guide the construction of similarity matrix, and encouraged by 31 , they chose the gradient attention to optimize the network. Although unsupervised hashing retrieval faces great challenges due to without labels information, these methods contribute to the development of image retrieval. Supervised hashing. Compared to unsupervised hashing, supervised hashing methods try to explore data labels as supervised information to calculate similarity matrix. Early on, Xia et al. proposed 32 to learn semantic features and hash codes separately, and there is no feedback between them. Recent supervised hashing usually designs an end-to-end learning framework to learn features and hash codes simultaneously such as [31][32][33][34] . On this basis, Cao et al. 18 selected a tanh activation function that make the network output is continuous hash codes. To avoid the discrete limit imposed on like-binary codes, Su et al. proposed 35 the greedy rules by updating the parameters toward the possible optimum discrete solution. In order to solve the problem of imbalanced distribution of data labels, Jiang et al. 36 introduced a soft concept that quantified pairwise similarity as a percentage by using labels information. Meanwhile, Cao et al. 37 proposed to weight the similarity matrix of training pairs and the Cauchy distribution is utilized instead of sigmoid function to calculate the loss. These methods are improvements in the loss function, but they ignore the problem of insufficient image features extraction. Hence, Li et al. 19 embedded channel attention and spatial attention into CNN to obtain sufficient semantic features. Yang et al. 34 improved the feature map in the dual attention module and combined it with the backbone network. However, these modules can aggravate the complexity of the network model and affect the speed of training. Motivated by 38 , this paper introduces a lightweight attention module based on ResNet18 and design a new class-wise loss, which suitable for learning more accurate hash codes.

Deep parameter-free attention hashing
In this section, the detail of DPFAH method is described, including research motivation, the definition of letters and formulas, the architecture of network, PFA module and the process of optimizing network. (1) shallow network cannot fully extract the semantic feature information of images, some channel-based or spatial-based attention modules can increase the complexity of model and lead to over fitting; (2) the process of relaxing hash codes can produce inevitable quantization error. In order to solve the problem of insufficient feature extraction, some scholars consider adding attention mechanism modules to the network, which will increase the complexity of network computing and algorithm time complexity. Based on the above considerations, the goal of this paper is to design a lightweight module that can extract image features without adding any parameters to the network, and a regulation term constrained hash codes is proposed to reduce the quantization error.
Problem formulation. In the similarity retrieval, given a dataset with n images are represented as where y i is the labels of the ith image and c is the number of classes. Therefore, the similarity matrix S = {s ij } is defined as: The target of deep hashing is to learn a hash function F(θ; x i ) that project x i to b i ∈ {−1, +1} l , where θ represents the parameters of CNN and l is the length of hash codes. Therefore, each image x i is mapped to l -dimensional vector U = {u i } n i=1 passing through F model, where u i is the l-dimensional vector of the ith image. To reduce quantification loss, inspired by 34 , u i is processed by a piecewise function as follow: Finally, b i = sign(f (u i )) is used to map l-dimension u i to l-bit b i , the sign(.) is defined as follow: Network architecture. Figure 1 shows the framework of DPFAH, which includes three main parts. DFPAH utilizes ResNet18 as backbone network, in order to fully improve the salient features representation ability and does not increase the computational complexity of the model. This paper has drawn a simple and parameter-free module into network, which can explore neurons in each channel or spatial location to learn more discriminative cues. In addition, the last layer of basic residual network is the classification layer that assigns data to the same class. On this basis, the hashing branch is designed parallel to the classification branch. The class-wise loss generated by the classification branch will positively affect the hashing branch when the parameters are updated by back propagation.
(1) s ij = 1, if x i and x j belong to the same class 0, otherwise www.nature.com/scientificreports/ PFA module. The existing attention modules consider the channel-wise attention or spatial-wise attention respectively. For channel-wise attention, the importance of each channel is firstly calculated from the perspective of channels, and then the channel with high importance is assigned greater 1-D weights. For spatial-wise attention, the importance of features at each location is calculated from a spatial point of view, and then the location with higher importance is assigned greater 2-D weights. These modules can increase the computational overhead when computing the 1-D or 2-D attention weights. Hence, this paper introduces a lightweight attention module (PFA) that can directly calculate 3-D weights. As shown in Fig. 2, first, the mean of X of feature maps X is obtained and calculate the square of X and X to get the variance. The variance is then divided by the feature map to obtain the variance of each channel, which is used to determine the variance of each channel and the importance of each spatial. Finally, the sigmoid function is used to restrict the result, and then multiplied with the original feature map X . In addition, the PFA module can focus on the primary areas close to the image label. As shown in Fig. 3, the second line represents the distribution of features extracted using the ResNet18 network, and the third line represents the PFA module is added to the network. The label of the first image is dog, only using the ResNet18 network to extract features will pay attention to many noises outside the label. After adding PFA, feature activations are mainly distributed around dog. It has the same effect on the second image. The third and fourth image focuses on more feature activations information about labels after adding PFA. Hence, the effectiveness of the PFA module is proved by the visualization of feature activation shown by Grad-CAM 39 . Thanks to the PFA introduces an energy function that derives a closed-form solution, it does not add parameters to the network. Inspired by neuroscience theories 40 , the neurons with the most information are usually the ones that show different firing patterns from those around them, and then those important neurons should be  www.nature.com/scientificreports/ given higher priority. The simplest means to discover these neurons is to compute the linear relationship between one target neuron and the others. Consequently, an energy function for each neuron is defined as follows: where t is target neuron and x i is surrounding neurons in each channel of feature map X ∈ R C×H×W , w t and b t are weight and bias, i is the ith spatial dimension, M is the number of neurons on a channel and M = H × W , t = w t t + b t and x i = w t x i + b t are linear transforms of t and x i . y t and y 0 is the output of target neuron and surrounding neurons respectively and y t = y 0 . The minimum value is gained by Eq. (4) when y t = t and y 0 = x i . In a channel, the linear separability between target neuron and other neurons can be obtained by calculating the minimum value of Eq. (4). For simplicity, this paper adopts y t = 1 and y 0 = −1 , add a regularization term to optimize the function. The energy function is transformed as follows: There are M energy functions on each channel, which are quite complex in calculation by using iterative. Luckily, Eq. (5) has a fast closed-form solution with respect to w t and b t as follows: represents the mean and variance of surrounding neurons, respectively. Thanks to the solutions of Eqs. (6) and (7) are calculated on a single channel. This supposes that all features in a single channel follows the same distribution. The mean and variances of all neurons can be computed according to this suppose. This method considerably reduces the calculation cost. Therefore, the minimal energy can be computed as follows: , it can be concluded that the greater the difference between the target neuron and the surrounding neurons, the lower the energy function and the more stable the model will be. Although e * t can represent the importance of each neuron, this method needs to calculate a large number of covariance matrix. Hence, this paper utilizes a scaling operator instead of an addition for feature refinement as follows: where E group all e * t across channel and spatial dimensions. Adding a sigmoid function to prevent the value of E from being too large.
Model formulation. Input a pair of images x i and x j into the network to generate hash codes b i and b j . The Hamming distance between b i and b j is defined as is the inner product and l is the length of hash codes. It can be seen that there are opposite changes between inner product and Hamming distance. The larger D H , the smaller b i , b j , and vice versa. Hence, the inner product is used instead of hamming distance to judge the similarity of pairwise images.
Given the set B = [b 1 , b 2 , . . . , b n ] of hash codes. The Maximum Likelihood estimation of B for dataset X is defined as follows: where P(S|B) represents the likelihood function. For each image pair, P s ij |b i , b j is the conditional probability of s ij under the given premise of b i and b j , which is calculated as follows: www.nature.com/scientificreports/ where σ (·) is sigmoid function defined as σ (x) = 1 1+e −x and b i = sign(u i ) . The reason why this paper uses u i instead of b i is that b i will cause a discrete optimization problem in Eq. (11). u i is the continuous like-binary codes output by the network, which can avoid this problem.
Learning hash codes by combing Eqs. (10) and (11) as follows: Equation (12) is the negative log likelihood loss function that shows the inner product of similar images should be as large as possible, the inner product of dissimilar images should be as small as possible. In other words, the hash codes of similar images are similar, and vice versa. Consequently, the hash codes preserve the similarity relation of the images in the original space.
In addition, there is an inevitable quantization error when u i is quantized to b i . To solve this problem, inspired by 9 , this paper has made the following improvements to u i : where ReLU(x) = max(0, x) is the Rectified Linear Unit. This paper follows the optimization policy proposed by 34 , which relax u i to [−δ, δ] and δ is set to 1.1.
Finally, in the classification layer, the output nodes of the network are determined by c that is the number of categories in the dataset. The loss between the output of the classification layer and the label y i is defined as: Additionally, o i is the real-valued classification layer outputs of the ith image. By calculating Eq. (14), the generated hash codes by hashing layer saves classification information at the same time.
Overall, combing Eqs. (12), (13) and (14), the total loss of the framework model is expressed as: Learning. The network parameters are optimized by calculating the gradient of the loss function and completing the back propagation. To learning a hash function for mapping images to hash codes, θ stands for the parameters of all feature layers, ϕ(x i ; θ) denotes the output of network, W T ǫR 512×l is the transpose of the weight matrix and v ∈ R l×1 represents bias vector. A fully connected layer is employed to connect feature representation and hashing learning. It is set: In the DPFAH model, the parameters to be optimized are θ , W , v and b i . The control variables method is adopted to optimize the parameters. Among them, b i can be directly optimized: Before optimizing the parameters θ , W and v , this paper calculates the derivative of L all with respect to u i and o i by Eq. (15) as: where, Then, this paper updates the parameters W and v by using back propagation: www.nature.com/scientificreports/ When optimizing network parameters, l 3 has a certain impact on parameter during back propagation, according to Eqs. (18) and (20), the gradient of θ is calculated as: The training process of the DPFAH model is exhibited in Algorithm 1.

Experiments
In this section, the DPFAH model is measured on three datasets. This paper compares the evaluation indexes of the DPFAH with the latest approaches.

Datasets.
(1) CIFAR-10 is a single-label public dataset, which include 60,000 images belonging to 10 classes, and each class have 6000 images. In this experiment, the training set is composed by selecting 500 images at random in each class, the testing set is formed by 100 images in each class. The remaining images are treated as the database. (2) NUS-WIDE is a multi-label public dataset including 269,648 images, this experiment selects 195,834 images belonging to 21 categories from them. Specifically, 100 images from each category form the testing set and the rest of images serve as the dataset. This experiments randomly select 500 images in each class as training set from the dataset. (3) Imagenet-100 is a single-label public dataset with 138,503 images and each image belongs to one of 100 classes. In experiment, the testing set is formed by 5000 randomly selected images, and the rest of the images serve as the database. At the same time, 130 images from each class of the dataset are chose as training set. In addition, the above three datasets are open-source datasets. All the procedures were performed in accordance with the relevant guidelines and regulations.
Evaluation metrics and settings. There are four evaluation metrics in the experiment to measure the performance of DPFAH: mean average precision (mAP), precision-recall curves (PR), precision curves within Hamming distance 2 (P@H = 2) and precision curves of the first 1000 retrieval results (P@N). In addition, this paper selects mAP@ALL for CIFAR-10, mAP@5000 for NUS-WIDE and mAP@1000 for Imagenet-100. In order to prove the performance of DPFAH, the methods of DBDH 14 , DSDH 5 , DHN 4 , LCDSH 6 , HashNet 18 , IDHN 7 , DFH 13 and DSH 3 are selected for comparative experiment. www.nature.com/scientificreports/ To make the experimental results objective and impartial, all comparative experiments are carried out on ResNet18 network and the Pytorch framework. Moreover, the parameter information of ResNet18 in each layer is shown in Table 1. Specifically, p is the size of the convolution kernel, s and k represent the stride and padding, respectively, and l is the length of hash codes.
In experiment, all comparative approaches use the same training set and testing set. The optimizer uses the root mean square prop (RMSProp), the mini batch size is set as 128, the learning rate is set as 5 × 10 −5 and the weight decay is set as 1 × 10 −5 . The environment configuration is shown in Table 2.
Hyperparameter analysis. In Eq. (15), this paper uses two hyperparameters η and ζ to weigh the impact of classification loss and quantization loss on network optimization. The values of η and ζ are determined by experimental results, as shown in Tables 3 and 4. This paper selects single-label dataset CIFAR-10 and multi-label dataset NUS-WIDE for parameter adjustment. Experiment fixes η = 10 when adjusting ζ . Similarly, experiment fixes ζ = 0.1 when adjusting η.
As shown in Table 3, the value of mAP is the largest on the two datasets when ζ = 0.1 . The mAP decreases significantly when ζ = 0.05 on CIFAR-10, and mAP on NUS-WIDE is also decreasing slightly. Compared with ζ = 0.1 , the mAP value of ζ = 0.5 decreased by 2.3% and 0.8% on average respectively on CIFAR-10 and NUS-WIDE. When ζ = 1, map values decreased by an average of 2.4% and 1.8% on two datasets. Therefore, it is concluded that when the hyperparameter ζ of class-wise loss is 0.1, the experimental result is better.

Layer Configuretion
Conv1 Hashing Layer l , the length of hash codes  Table 3. mAP for of different ζ.  Table 4. mAP for of different η. www.nature.com/scientificreports/ Figure 3 shows mAP on different ζ more intuitively, the mAP curves reach the peak when the value of a is 0.1. As the value of ζ becomes larger or smaller, the value of mAP will decrease slightly. Therefore, this paper chooses ζ = 0.1 to achieve the optimal experimental effect.
As shown in Table 4, When η = 10 , mAP reaches its maximum value. On CIFAR-10 and NUS-WIDE, compared with η = 10 , the mAP value of η = 1 decreased by 1.9% and 0.3% on average, the mAP value of η = 5 decreased by 2.1% and 0.1% on average, and the mAP value of η = 15 decreased by 2.0% and 0.6% on average respectively. Therefore, when the hyperparameter η of quantization loss is 10, good results can be obtained in the experiment.
Similarly, as shown in Fig. 5, the value of mAP is higher than the others when η = 10 , and the mAP curves reach the peak on CIFAR-10. On NUS-WIDE, when η takes 5 and 10, the mAP at 48 bit and 64 bit are close, but the mAP value of η = 10 is significantly better than η = 5 at 16 bit and 32 bit. Therefore, this paper also sets η as 10 to achieve optimal experimental effect.
Empirical analysis. In order to fully extract image features without increasing network complexity, this paper adds the PFA module to ResNet18 network, which can extract 3-D weights of features. Compared with the common attention mechanism module, the structure of PFA is simple and parameters-free. Meanwhile, to improve the discrimination and accuracy of hash codes, this paper designs classification branches in the network. Equation (13) is designed to reduce quantization errors. Equation (14) is the class-wise loss generated by the classification layer. Equations (13) and (14) are integrated to L loss in ablation experiments. As shown in Table 5, DBDH is selected as the baseline and the length of hash codes is 48 bit on CFIAR-10 dataset. DBDH indicates the baseline model utilizing AlexNet network. DFPAH-1 chooses ResNet18 as backbone instead of  www.nature.com/scientificreports/ AlexNet. On this basis, DFPAH-2 shows that PFA Module has been added to the network. DFPAH-3 adds L loss to the network. The symbol √ indicates adding corresponding module. As shown in Table 5, PFA module is added on the basis of DFPAH-1, and the mAP value is increased by 2.95%, which proves that PFA module improves the accuracy of image retrieval. The mAP value of DFPAH-3 is up to 0.98% higher than DFPAH-2, showing the effectiveness of L loss . Figure 6a intuitively shows the PR curves added with PFA module and L loss , which is significantly higher than the baseline model. Figure 6b displays the precision of returning the first 1000 images, DFPAH-3 is obviously better than others. Hence, the above ablation experiments verify the effectiveness of PFA module and L loss . Figure 7 shows the t-SNE Visualization of the hash codes learned by DPFAH and the baseline DBDH on CIFAR-10 dataset. As shown in Fig. 7a, the hash codes generated by DPFAH show clear discriminative structures where the hash codes in different categories are well separated, while the hash codes generated by DBDH do not show such clear structures. This verifies that by introducing the PFA module and L loss for hashing, the hash codes generated through DPFAH are more discriminative than that generated by DBDH. Therefore, DPFAH method effectively increases the spacing between inter classes and reduces the gap intra classes, making the generated hash codes compact and effectively enhancing the representation ability. The curve of PR is an evaluation index with precision and recall as variables. Recall in the curve is set as the abscissa and precision is set as the ordinate. If the PR curve of one algorithm is completely surrounded by another algorithm, it can be asserted that the performance of the latter is better than that of the former. Therefore, the performance of the algorithm is judged by the area enclosed by the PR curve. Figure 8 shows the PR curves on www.nature.com/scientificreports/ dataset CIFAR-10. As can be seen from the Figure 8a-d, the curves of DPFAH method are significantly higher than all comparative methods. In particular, when the length of hash codes is 16bit, the enclosed area is much larger than DSDH, which has the best performance among all comparative methods. Because NUS-WIDE is a multi-label dataset and the calculation process is relatively complex, the improvement on NUS-WIDE is not as obvious as that on CIFAR-10, but it is still the best of all methods. As shown in Figure 9, the mAP of DPFAH is the highest compared with the other eight comparison algorithms. In Figure 9a-d, DPFAH is higher than DSDH that has the best performance among all methods. Figure 10 shows the PR curves of 16, 32, 48 and 64 bits on Imagenet-100 dataset. The PR curve of DPFAH method is significantly higher than that of other comparison methods, especially on Fig. 10b-d In Fig. 10a, when recall is greater than about 0.6, the precision of DPFAH is less than that of DHN. When recall is less than 0.6, the precision of DPFAH is much higher than that of DHN. It can be seen from the overall PR curve siege area that DPFAH is significantly greater than DHN.

Results analysis. As shown in
To achieve the aim that the hamming ranking only needs O(1) time searches, the evaluating indicator P@H = 2 is important for the retrieval of hash codes. Figure 11 shows the result of P@H = 2 on three datasets, the method DPFAH obtains the highest precision in experiment. With the increase of hash code length, the precision also increases steadily, which shows that DPFAH model is more stable than the methods of DSH, IDHN and DHN on CIFAR-10, NUS-WIDE and Imagenet-100.
Another evaluation metric is the curves of P@N. The precision of the first 1000 images are selected in this experiment. Figure 12 shows the result of P@N on CIFAR-10 dataset, DPFAH method has achieved better precision than the other methods. Specifically, in Fig. 12a, the curves P@N of DPFAH is significantly higher than DHN and DSDH. In Fig. 12b-d, although the growth rate of DPFAH is not as obvious as Fig. 12a, the best precision is still obtained on 32bit, 48bit and 64bit.   www.nature.com/scientificreports/ Figure 13 shows the P@N curves on NUS-WIDE, as can be from Fig. 13a,b, the P@N curves of all methods is relatively stable with the number of returned images increases. Compared with other algorithms, DPFAH still achieves the highest precision. Figure 14 shows the P@N curves on the Imagenet-100 dataset. As can be seen from Figure 14b-d, when the length of the hash codes is 32, 48 and 64bits, the effect of DPFAH is obviously better than the other methods. With the increase of the number of images, the precision shows a stable trend, but the best results are still obtained in all comparison algorithms. Fig. 15, this paper visualizes the top 10 returned images of DPFAH for eight query images on Imagenet-100. The first row shows the label of the query images, the second row is query images, the retrieval results of DPFAH are shown at other rows. The red boxes are used to mark the false retrieval results.

Conclusions
Existing image retrieval methods based on deep hashing have the defects of imbalance and insufficiency when existing hashing methods extract image features. Some scholars propose to employ channel-wise or spatial-wise attention mechanism into the network, which will add many parameters to the model and increase the computational complexity. Hence, this paper introduces a PFA module and propose DPFAH method. PFA module based on well-established suppression theory and define an energy function that determine the importance of each neuron. This module does not add any parameters to the network and directly extracts 3-D weight information of feature map. In addition, to generate accurate hash codes that retain the similarity information of the original image, this paper designs a classification branch to optimal network. The effectiveness of DPFAH method is proved by a large number of experiments. In particular, the evaluation index mAP increased by 2.95% when the PFA module is added in network. Hence, a better image retrieval model is obtained by DPFAH method. www.nature.com/scientificreports/